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@> Method of synthesizing optical images. 

(§) A method and apparatus for forming a syn- 
thesized optica! image with al! of the desired 
objects in focus. A prescribed number of optical 
images of the same scene are provided by 
recording an optical image at a respective focal 
distance and changing the focal distance for 
each additional optica! image as recorded. The 
optical images are subjected to wavelet trans- 
formation to form a multi-resolution represen- 
tation. The coefficients of the various 
multi-resolution representations are compared 
at the same position to detect the maximum 
spectral amplitudes. Based on the detected 
maximum spectral amplitudes, another multi- 
resolution representation is obtained. Then, an 
inverse wavelet transformation is performed to 
obtain the synthesized optical image. 
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This invention concerns a method for synthesiz- 
ing optica! images. More specifically, this invention 
concerns a method for synthesizing multiple optical 
images obtained by changing the focal distance. 

BACKGROUND OF THE INVENTION 

When a camera is used to record optical images 
and objects relatively far from the camera lens are in 
focus, objects nearer to the camera lens are out of fo- 
cus and their recorded images are blurred. On the 
other hand, when focusing on the nearer objects, the 
farther objects are out of focus and their recorded im- 
ages are blurred. This problem is believed to be relat- 
ed to the intrinsic characteristics of the optical sys- 
tem. According to a basic optical principle, a pinhole 
lens has an infinite focal depth, and all objects are in 
focus when their images are recorded within the pin- 
hole lens of a pinhole camera. However, for the [ideal] 
pinhole lens, the hole is so small that not even a single 
photon can pass through it, and hence no image is 
formed. That is, it is merely a hypothetical lens. 

Due to this basic physical limitation, the lens has 
a finite focal depth, and the picture of the object is tak- 
en by means of automatic or manual focusing. How- 
ever, when two objects, one nearer the camera and 
the other farther away from it, are to be photographed 
in the same picture, it is impossible to bring both into 
focus at the same time. That is, while bringing one ob- 
ject into focus, the other goes out of focus and its im- 
age becomes blurred. This is a disadvantage. 

Although not a technology for forming a synthe- 
sized image, a technology known as inverse filtering 
has been tested which attempts to create a clear im- 
age from the blurred portion of a picked-up image by 
means of enhancing the blurred portion. However, 
since it also amplif ies noise, it fails to reach the stage 
of practical applications. 

OBJECT OF THIS INVENTION 

An object of this invention is to provide a method 
for forming a synthesized image in which multiple opt- 
ical images picked up at different focal distances are 
in focus in the synthesized image. 

SUMMARY OF THE INVENTION 

This invention provides a method for obtaining a 
synthesized image while bringing objects at different 
positions into focus by means of digital signal proc- 
essing using multiple image representations. Accord- 
ing to one aspect of the invention there is provided a 
method for synthesizing transformations from vari- 
ous image element signals of multiple optical images 
added at different focal distances to obtain multi-re- 
solution aforementioned optical images; comparing 
the various coefficients of the aforementioned optical 



images at the same position so as to detect the max- 
imum spectral amplitude detected at each of the 
aforementioned prescribed positions; and forming the 
image element signals from the aforementioned syn- 
5 thesized multi-resolution 

According to another aspect there is provided im- 
aging apparatus for forming a synthesized image 
from all objects in a field of view (FOV) comprising:- 
image capturing means capable of forming a focused 
id image of one or more objects in the field of view; 
memory means for stored the focused images; and 
image synthesizing means for synthesizing an image 
of the or each object in the field of view from the stor- 
ed focused images. 
15 Reference will now be made by way of example 

to the accompanying drawings, in which:- 

Fig 1 is a flow chart illustrating an embodiment of 
a method for synthesizing optical images in accor- 
dance with the present invention. 
20 Fig 2 is a block diagram illustrating an embodi- 

ment of an apparatus for implementing the method of 
synthesizing optical images in accordance with the 
present invention. 

Figure 3 is a block diagram illustrating another 
25 embodiment of an apparatus for implementing the 
method for synthesizing optical images in accor- 
dance with the present invention. 

Figure 4 is a schematic diagram illustrating an 
embodiment for implementing multi-resolution repre- 
30 sentation in accordance with the invention. 

Figure 5 is a photograph taken with a camera lens 
system picking up and focusing on a nearby object. 

Figure 6 is a photograph of the same scene 
shown in Figure 5, but with the camera lens system 
35 picking up and focusing on a remote object. 

Figure 7 is a diagram illustrating a multi-resolu- 
tion representation of the optical image shown in Fig- 
ure 5 as an embodiment of this invention. 

Figure 8 is a diagram illustrating a multi-resoiu- 
40 tion representation of the optical image shown in Fig- 
ure 6 as an embodiment of this invention. 

Figure 9 is a synthesized photograph of the same 
scene shown in Figures 5 and 6, but illustrating a syn- 
thesized optical image in accordance with the present 
45 invention. 

Figure 10 is a diagram illustrating a portion of the 
program in an embodiment of the forward transforma- 
tion of a multi-resolution representation in accor- 
dance with the invention. 
so Figure 11 is a diagram illustrating a portion of the 

program in an embodiment of the forward transforma- 
tion of a multi-resolution representation in accor- 
dance with the invention. 

Figure 12 is a diagram illustrating a portion of the 
55 program in an embodiment of the inverse transforma- 
tion of a multi-resolution representation in accor- 
dance with the invention. 

Figure 13 is a diagram illustrating a portion of the 
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program in an embodiment of the inverse transforma- 
tion of a multi-resolution representation in accor- 
dance with the invention. 

In reference numerals as shown in the drawings: 

10, image device 

20, pickup device 

30, synthesis device 

40, frame memory 

I, synthesized image 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

Figure 1 is a flow chart illustrating an embodi- 
ment of a method for synthesizing an optical image in 
accordance with the invention. First of all, multiple 
optical images are formed at respective different fo- 
cal distances by changing the focal distance of the 
camera lens (step 1). it is possible to adjust the focal 
distance either linearly or noniinearly in the range 
from infinity to zero. However, there is virtually no 
problem in specifying the focal distance itself. It is 
only necessary to focus to the extent to which all ob- 
jects to be brought into focus can be extracted. An ap- 
propriate number of images are picked up in this step. 

Then, the picked-up images are transformed to 
the mutti- resolution representation (step 2). This mul- 
ti-resolution representation is based on the multi-re- 
solution technology discussed recently in mathemat- 
ics (see: C. H. Chui: "An introduction to wavelets;" I. 
Daubechies: "Orthonormal bases of compactly sup- 
ported wavelets." Communication of pure and applied 
mathematics, Vol. XLI 909-996 (1988)). Generally 
speaking, it is an isolation technology, in which, by 
means of special filtering and subsampling, the orig- 
inal signals are isolated into high-frequency and low- 
frequency components without increasing the total 
amount of data. A prominent feature of this isolation 
technology is that the aforementioned low-frequency 
component can be further isolated using the same fil- 
tering method. In this way, the operation can be re- 
peated for the required number of cycles for the spe- 
cific application. The layered signal representation in 
the frequency region generated in this way is usually 
called multi-resolution representation. 

Then, the transformed multi-resolution represen- 
tations are synthesized to form another new multi-re- 
solution representation (step 3), followed by inverse 
multi-resolution representation to form the synthe- 
sized image (step 4). That is, the multiple multi-reso- 
lution representations corresponding to the images in 
focus at different distances are synthesized to form 
another multi-resolution representation, which is then 
subjected to an inverse transformation to obtain a 
synthesized image in which the various objects are 
optimally in focus. 

Figure 2 is a diagram illustrating an example of 
the image equipment apparatus for implementing the 



method of this invention. In this example, the method 
and equipment for forming a synthesized image I for 
which optimal focusing is realized for all of the ob- 
jects, from object A. nearest to the image equipment 

5 1 0 to object Z, farthest away from it. Pickup device 20 
first focuses on object A (with the shortest focal dis- 
tance) and its optical image (image l^ is recorded via 
a CCD (not shown in Figure 2) in frame memory 40. 
Then, the pickup device 20 focuses on object B (with 

10 the focal distance increased slightly), and its image 
is recorded in another frame memory (or on another 
region in the same frame memory). In the same way, 
the operation is performed in order until focusing of 
the most remote object Z (with focal distance up to in- 

15 f inity) is performed, and its image l 2 is recorded in the 
corresponding frame memory. (For the various ob- 
jects, there is no need to record A in l A , B in l Bl etc., 

they may be matched to l n (n = A Z) in any [order]. 

However, in the present example, it is assumed that 

20 A is focused to form l A , B is focused to form l B , ...) Fo- 
cusing for objects A-Z is carried out by means of the 
control signals from synthesis device 30. However, it 
may also be performed manually or by means of other 
control equipment. The order in which the images are 

25 picked up can be selected at will. The method to be 
explained later is used to synthesize images t A -l z re- 
corded in the various frame memories by means of 
synthesis device 30, forming synthesized image I. 
The memory may be in the form of a magnetic mem- 

30 ory or an optical memory. It is also possible to input 
the picked-up signals into synthesis device 30. Also, 
photoelectric conversion equipment other than a 
CCD may be used as well. 

Figure 3 illustrates another embodiment of the 

35 apparatus. It differs from the case shown in Figure 2 
in that it has multiple pickup devices. For simplicity, 
only two pickup devices 20 and '25 are shown in Fig- 
ure 3. The input optica! information is equally divided 
between half-mirror 40 and mirror 50, and the por- 

40 tions are input into pickup device 20 and pickup de- 
vice 25, respectively. Consequently, one of the pick- 
up devices 20 or 25 focuses on object A (with a shorter 
focal distance) to add image l A , which is then record- 
ed in frame memory 40. Then the other pickup device 

45 focuses on object B (with a longer focal distance) to 
capture image l B; which is recorded in frame memory 
45. Then, by using a method to be explained later, an 
image I, which corresponds to the optimal focusing 
for both objects A and B, is formed by means of a syn- 

50 thesis device 30. 

in the following, the multi-resolution representa- 
tion using the wavelet coefficients executed by syn- 
thesis device 30 will be explained. As shown in Figure 
4, image element signal A from the frame memory is 

55 isolated into four wavelet transformations a, b. c, and 
d by means of a hierarchical filter bank. Filters H and 
G correspond to high- pass and low-pass filters, re- 
spectively, and are complementary to each other 
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(mirror filters). Filters G and H are called wavelet fil- 
ters when f iltering and subsampiing (blocks indicated 
by arrows) to form the orthogonal isolation of the input 
signal. Figure 4 is a diagram illustrating the case of 
3-layer isolation of the signal by means of one-dimen- 
sional (in either the horizontal or vertical direction) fil- 
tering. For the 2-dimensional filtering adopted in this 
invention, sequential filtering is performed in the hor- 
izontal and vertical directions. That is, after execution 
of filtering in the horizontal direction (or vertical direc- 
tion), filtering in the vertical direction (or horizontal di- 
rection) is executed in this configuration. 

Although a frequency domain representation, the 
multi-resolution representation also has spatial infor- 
mation (information related to position). On the other 
hand, for the Fourier transform, although a good rep- 
resentation in the frequency domain can be realized, 
no spatial information can be preserved; hence, the 
position cannot be specified (the uncertainty princi- 
ple). 

Next, the synthesis of multi-resolution images by 
means of multi-resolution representation will be ex- 
plained. The multi-resolution representation using 
wavelet transformation is a representation of the in- 
put signal in the frequency domain. This representa- 
tion also represents spatial information at the same 
time. In the multi-focus synthesis method, N multiple 
input images recorded for the same scene followed by 
wavelet transformation are the objects, and the wa- 
velet coefficients that represent the most intense 
high-frequency component at each image element 
position are selected, since they are taken as the rep- 
resentation of the focal image at the aforementioned 
position. This process is executed for all of the image 
element positions of the image. By means of synthe- 
sis of the image elements selected for one multi-re- 
solution representation detected at each position, a 
new multi-resolution representation can be formed 
with all of the image elements optimally focused. If 
needed, the aforementioned process may be not im- 
plemented for the lowest position at which the wavelet 
coefficients have almost the same value. 

On the other hand, for the aforementioned syn- 
thesis, it is necessary to evaluate the spectral distrib- 
ution of the image by changing the focal distance. 
Consequently, the Fourier transform of the overall im- 
age or partial small blocks (such as the cosine trans- 
form method used as a basic technology of image 
compression) may be considered. However, in this 
approach, the spectral lines and ends of lines usually 
expand into multiple coefficients, and this phenom- 
enon-makes the frequency and position information 
unclear. Consequently, in the wavelet transformation, 
these two types of information are preserved, and 
hence it is basically appropriate for use in the multi- 
focus synthesis method. 

Figure 5 is a photograph which illustrates the opt- 
ica! image provided by focusing on a nearby cup. Fig- 



ure 6 is a photograph which illustrates the optical im- 
age provided by focusing on a remote tree in the 
same scene shown in Figure 5. Figures 7 and 8 show 
the multi-resolution representations for the images of 
5 Figures 5 and 6, respectively. For Figures 5 and 6, the 
original image is represented by means of 704 pixels 
x 480 lines. They are isolated into 5 frequency bands 
shown in Figures 7 and 8. in the 2-dimensional multi- 
resolution representation, the lower right corre- 
w sponds to the higher frequency components. The 
lowest layer (the upper left corner) corresponds to the 
lowest frequency components, and it can be taken as 
the image which can pass through the lowest low- 
pass filter (and subsampied). 
15 In the case of mult'hfocus synthesis, the scenes 

of the same image are photographed for the desired 
number of cycles by a camera (or multiple cameras) 
at different focal distances. However, for the iowest 
layer generated by the lowest low-pass filtering oper- 
20 ation, even when photographing at different focal dis- 
tances, the images obtained are still almost identical. 
(In other words, it is preferred that the number of lay- 
ers be set to make the lowest frequency layers almost 
identical). The lowest frequency layer represents the 
25 mean brightness of the image. 

On the other hand, for the higher frequency lay- 
ers, the higher spectral components appear at vari- 
ous positions in the multi-resolution representation, 
and they correspond well to the objects in focus at the 
30 various focal distances. 

Consequently, when the multi-resolution repre- 
sentation is adopted, the following takes place: 

(1) In the lowest frequency layer (the 15 pixel x 
22 lines on the upper left corner in Figures 7 and 
35 8), each coefficient is derived as an average of 

multiple input images, or by extraction from the 
coefficients of a prescribed image. It is used as 
the lowest frequency layer of the synthesized im- 
age. 

40 (2) For the upper layers, at each position, there 

is a multi-resolution representation that gives the 
maximum spectral amplitude (maximum abso- 
lute value). That is, for the multi-resolution repre- 
sentation that gives the maximum spectral ampli- 

45 tude at position of i, j, as the simplest method, 

max( I \j< 1 > I, lAi.pl k ti (N >l) (1) 

is used as the representation (N = number of the 
multi-resolution representation, that is, number of 
the optical images added). It corresponds to the 

50 optimal focusing of the input scene, and the cor- 

responding multi-resolution coefficients give the 
information. The overall multi-resolution repre- 
sentation is formed according to this rule, in this 
way, at ail of the positions, the multi-resolution 

55 representations that give the maximum spectral 

amplitude are detected, and they are synthesized 
to form the new multi-resolution representation. 
This rule itself is believed to have various forms. 
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Then, based on the detected maximum spectral 
amplitude position, an inverse multi-resolution trans- 
formation is performed so that the synthesized multi- 
resolution representation is transformed to an optical 
image composed of image element signals. As shown 
in Figure 9, this optical image is a synthesized image 
of a nearby cup and a far away tree, both of which are 
well in focus in the picture. 

Although not directly related to this invention, de- 
pending on the method of synthesis, there are cases 
in which the frame sizes may differ. This is an influ- 
ence of the zoom effect as related to focusing of the 
various optical images. If needed, the size may be 
changed using various adjustment methods, such as 
"enlargement," "contraction," "interpolation," "extrac- 
tion,*' etc., before the synthesis operation. 

Figures 10-13 illustrate portions of the program 
for the forward and inverse wavelet multi-resolution 
transforms. 

The aforementioned synthesis method does not 
depend on the image techniques related to content, 
such as edge extraction, texture analysis, etc. The 
present method for synthesizing is completely differ- 
ent from methods with any precondition related to 3- 
dimensional objects. It is a method in which a stable 
synthesized result can be obtained with respect to ail 
natural images. (That is, there is absolutely no gen- 
eration of significant degradation in the case when 
the precondition is violated.) tn addition, the edge 
analysis is still in the stage of initial development. 
When the number of edges is increased, the process- 
ing time increases exponentially. Consequently, it 
has not been used in practical applications. At pres- 
ent, the present method is the most appropriate meth- 
od for practical applications for natural images. 

Also, since it is based on detection of the maxi- 
mum spectral amplitude from the multiple multi-reso- 
lution coefficients, there is no limitation to the number 
of optical images which can be added. By capturing 
multiple optical images, it is possible to obtain a syn- 
thesized image that is in focus at almost all positions. 
Also, enlargement is possible to obtain very correct 3- 
D information (distance, appearance of object, etc.) 

tn the above description, embodiments of this in- 
vention have been presented. However, this invention 
is not limited to these embodiments. This invention 
can be used with various types of equipment, such as 
cameras, television cameras, monitoring cameras, 
medical imaging cameras, etc. 

This invention provides an effective method for 
forming a single synthesized optical image from mul- 
tiple optica! images added at different focal distanc- 
es, with optimal focusing. 

Claims 

1. A method for synthesizing optical images com- 



prising of the following steps: performing wavelet 
transformations from various image element sig- 
nals of multiple optical images added at different 
focal distances to obtain multi-resolution repre- 

5 sentations with a number corresponding to the 

number of the aforementioned optical images.; 
comparing the various coefficients of the afore- 
mentioned multi-resolution representations cor- 
responding to the aforementioned optical images 

w at the same position so as to detect the maximum 

spectral amplitude at each of the prescribed pos- 
itions; and forming the image element signals 
from the aforementioned synthesized multi-reso- 
lution representation by means of inverse wave- 

15 let transformation. 

2. imaging apparatus for forming a synthesized im- 
age from all objects in a field of view; memory 
means for storing the focused images; and image 

20 synthesizing means for synthesizing an image of 

the or each object in the field of view from the 
stored focused images. 

3. The Apparatus of claim 2, wherein the image syn- 
25 thesizing means comprise one or more filters. 

4. The Apparatus of claim 3, wherein the filters form 
a hierarcica! filter bank comprising low-pass and 
high-pass filters. 

30 

5. The apparatus of any claims 2 to 4, wherein the 
image capturing means comprise a single device 
and image capture of each object occurs sequen- 
tially. 

35 

6. The Apparatus of claim 5, wherein the image cap- 
turing means captures the image from the clos- 
est object first and the farthest object last. 

40 7. The Apparatus of any of claims 2 to 4, wherein the 
image capturing means plurality of devices each 
adapted to capture the image of a different object 
in the field of view. 

45 8. The Apparatus of any of claims 2 to 7, wherein the 
memory means comprise a frame memory for 
storing the image formed from the or each image 
capturing means. 

so 9. > Apparatus for carrying out the method of claim 1 . 
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